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Presented is an operational, comprehensive plan for 
measuring a Competency Based Teacher Education (CBTE) program. This 
assessment and evaluation system is a part of the competency-based 
program of Vocational and Applied Arts Education (VAE) • Its seven 
major goals are: (1) to provide objective public measures for each 
performance objective in the program; (2) to provide for field 
assessment of intern teachers and pre-interns using classroom 
teachers as the primary evaluators; (3) to identify student skill 
deficiencies prior to instruction; (4) to provide means by vhich 
students can be exempt from unneeded instruction; (5) to provide 
means by vhich student reactions can be systematically gathered for 
use in revising instruction and instructional materials; (6) to 
provide means by vhich data can be collected to give direction for 
program revision and l2prv'>vement ; and (7) to determine the effects of 
the program upon the graduates* teaching and upon their pupils. The 
assessment and evaluation system is perceived as the most crucial 
aspect of CBTE. Procedures for determining test reliability, test 
item validity instruments, a pre-intern field experience check list, 
data collection instruments and reports available from the Assessment 
Data Bank are appended. (Author/BJ6) 
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INTRODUCTION 

The overriding problem [vn'th CBTe] before 
which the others pale to insignificance is 
that of the adequacy of measurement instru- 
ments and procedures.' 

I assume that even small progress made in 
assessing teacher competence will be of 
great improvement over our present 
evaluations/ 

These two positions reflect the basic philosophy upon 
which the assessment and evaluation model described in this 
paper has been built and is now presented. It is true that 
the problem of measurement in CBTE programs is critical; it 
creates a vulnerability— philosophically, politically and 
practically. The instruments used in competency-based 
programs, hov/ever, are no worse than those used in traditional 
programs which currently are the basis for certification 
recommendations. All efforts which are put into devising 
valid, reliable assessment and evaluation systems are steps 
leading toward valid teacher training programs. The descrip- 
tion here is of one approach; clearly, it is not a perfect * 
one, but it is a working model. 

This assessment and evaluation system is a part of the 
competency-based program of Vocational and Applied Arts 
Education (VAE). This is a unit in the Division of Teacher 



El am, Stanley, Performance-Based Teacher Education; The 
State of the Art , Performance-Based Teacher Education Series: 
No. 1 (Washington, D. C: American Association of Colleges for 
Teacher Education, 1971), p. 21. 

^McDonald, Frederick J., "The State of the Art In Performance 
Assessment of Teaching Competence," Assessment (Albany, New York: 
Multi-State Consortium on Performance-Based Teacher Education, 1974), 
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Education in the College of Education at Wayne State University 

in Detroit, Michigan. The VAE undergraduate program includes 

approximately 500 students on the active role. This CBTE 
3 

program was put into operation in 1972 after a year of develop- 
ment. However, the program revision and growth still continues. 
A major concern and focal point of activity is the matter of 
student competency assessment as a part of both campus and 
field experiences and program evaluation. This model will 
encompass : 

1. the design and use of instruments to assess 
student performance 

2. efforts to establish reliability and validity 
of these instruments 

3. assessment data collection and resulting reports 

4. formative program evaluation techniques 

5. program follow-up design. 

Assumptions 

The ass ssment and evaluation model rests upon some key 
assumptions: 

1. The ultimate validity of a CBTE program must be 
determined by identifying the effects of the VAE 
graduates upon their own pupils. 

2. The use of the systems approach will lead to 
continually-improved programs and program components. 



For a complete program description, see "A Working Model 
of a Competency-Based Teacher Education Program" by Fred S. Cook, 
Charlotte Neuhauser and Rita C. Richey, a paper presented at the 
annual meeting of the American Educational Research Association, 
New Orleans, February, 1973. 
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3. Comprehensive research will be needed to identify 
effective teacher competencies. Pending this data, 
teacher education programs should be based upon 
local data from the professors, former graduates, 
cooperating classroom teachers, the community and 
current students. 

4. Classroom teachers should be involved in the 
development and implementation of the assessment 
and evaluation part of a CBTE program, just as 
they are involved in other facets. 



Goals of the Assessment and Evaluation System 

The major objectives of a specific assessment and 
evaluation system reflect, of course, the design of the total 
program, as well as the priorities of the developers. In this 
particular case, there are seven major goals: 

1. To provide objective public measures for each 
performance objective in the program. 

2. To provide for field assessment of intern teachers 
and pre-lnterns using classroom teachers as the 
primary evaluators. 

3. To identify student skill deficiencies prior to 
instruction. 

4. To provide means by which students can exempt 
from unneeded instruction. 

5. To provide means by which student reactions can « 
be systematically gathered for use in revising 
instruction and instructional materials. 

6. To provide means by which data can be collected 
to give direction for program revision and 
improvement. 

7. To determine the effects of the program upon the 
graduates* teaching and upon their pupils. 
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DESIGNING AND IMPLEMENTING 
OBJECTIVE-REFERENCED TEST ITEMS 



General Assessment Framework 

In the VAE competency-based program there are three key 
testing points: 

1. The exemption test is given prior to instruction 
in a specific course. This test provides the 
means by which each student can demonstrate his 
mastery of each terminal performance objective 
for that phase of the program. ^ 

2. The prerequisite skills test is given prior to 
instruction in each course. Each test requires 
students to demonstrate their mastery of the 
skills which are necessary to be successful 
for that particular phase. 

3. The exit test is given after instruction and, 
like the exemption test, is the vehicle by which 
students demonstrate their mastery of that phase's 
terminal performance objectives. 

This three-faceted approach to student assessment is 

followed in each phase of the program, as well as in any 

instructional modules which might be used. The only deviation 

to this is in the Intern teaching exemption test. While it is 

possible to take this test, it is not administered on a regular 

basis because of two constraints: the length of time to 

administer the exam (ca. 4 weeks) and the need for a cooperating 

field site. Typically, a person who would be approved to take 

the test would be an individual who is currently teaching under 

an emergency or special certificate. 



The VAE Pre-certification Program consists of four phases 
taken In sequence: 1) Introduction to Education (4 qtr. nrs.}, 
2) Foundations of Education (Ec^ucational Psychology and Analysis 
of Teaching combined for 8 qtr. hrs.), 3) Methods of Teaching 
(8 qtr. hrs.), the Intern Teaching block (16 qtr. hrs. of student 
O teaching, a 4-hour seminar and 4 hours of Educational Philosophy). 
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Principles of Item Construction and Use 

The following principles of test item construction apply 
to items used for any of the three assessment purposes. These 
principles have evolved out of experience, conviction and 
practical constraints. The major principles are: 

1. All items will be objective-referenced. 

2. All items must be approved by the faculty 
before use. 

3. The development processes must allow for 
immediate use and continued refinement. 

4. The administration of the tests, data collection, 
reliability and validity analysis must be possible 
to complete with average university resources and 
expertise. 

5. Test items may and should utilize a variety of 
modes. Acceptable modes Include paper and pencil, 
product evaluation, performance demonstration, etc. 
The goal is for performance and product items. 

6. All test items, regardless of mode, should conform 
to one format. 

7. All items throughout the program on a similar topic 
should have a congruity in terms of criteria and 
approach. 

8. If the test construction process shows a weak 
performance objective Ce.g., inadequate criteria, , 
lack of congruity with similar performance 
objectives), the performance objective should be 
changed before the test item is finalized. 

9. The performance items must be possible to administer 
within a public school during normal school hours, 
normal class periods or in a simulated laboratory 
setting. 

10. The equipment needed to administer the tests must 
be confined to those items normally available in 
an urban high school . 
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n. The test items must be written so that they can be 
administered by a classroom teacher with a Master's 
degree, an average of five years of teaching 
experience and three hours of training provided 
by the department OR a college supervisoi^ who has 
had three hours of departmental training. 

12. All test items are public documents. (See pagelO 
for further discussion.) 

These principles provide for implementation in the public 

schools, construction without relying on unrealistic resources, 

standardized format and faculty approval. Not following any 

one of these guidelines will probably lead to probl ems— probl ems 

with your public school constituency, problems with your faculty 

who must use the tests and problems with youir students who must 

understand the tests. 

Test Item Format 

A test item has been described by Hively as "a set of 
instructions telling how to (1) set up a specific problem 
for someone, (2) record his response, and (3) score the results."^ 
The format of test items in a competency-based curriculum presents 
more of a problem than that of the format of a traditional test. 
First, there are an abundance of performance items, as opposed 
to typical multiple-choice items. Such performance, to as great 
an extent as possible, should be off -campus in the public schools. 
Being an actual demonstration of skill, the scoring facets may 
well be quite complicated with, perhaps, several persons involved. 



^Hively, Wells. "Domain-Reference Testing," mimeographed 
paper, February, 1973, p. 6. 
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All of these factors present complications in how the item should 
actually be written on the paper. 

Next, the test item must be constructed in such a manner as 
to facilitate the understanding of all processe-— the performance 
expected, the scoring and precise criteria which will be applied, 
as well as any preparations which must be made in advance. 
Students and test administrators alike can become completely 
frustrated when given a stack of papers seemingly without end 
and a maze of directions.^ 

FIGURE 1 (see next two pages) is a test item used in the 
intern teaching phase of the VAE Pre-certification Program.^ 
You can examine the format and identify the following components: 

1. The related performance objective 

2. The directions to the inte rn 

This includes a brief description of the evaluation 
process and, if appropriate, a list of steps the 
student must follow to complete the test item. 

3. The directions to the evaluator 

This includes any steps necessary to prepare for 
the measurement, as well as the precise evaluation 
procedures. The most important part of this section, 
however, is the exact scoring guidelines. These are 
the specific questions one v/ould ask oneself as all, 
facets of the performance are examined. With each 
question are the specific criteria one must meet 
to pass that particular component of the item. 

4. The indication of mastery 

This includes a verbal summary of the criteria 
for passing and a spot to check if the item was 
passed or if the student must recycle. 



^Rita C. Richey, Designing a CBTE Instructional System: A 
VAE Case History , Competency-Based Teacher Education Series No. 3 
(.Detroit, Michigan: Vocational and Applied Arts Education, 1974), 
pp. 18-19. 

^The VAE Intern Teaching Examination was developed criginally 
by Effective Feedback, Inc. in consultation with the VAE faculty |Q 
and project staff. The entire exam has subsequently been revised 
by Kathleen M. Herschelmann after extensive try-out and input 
from the users. 



0bjt:ctive 09808 FfGimE 1 ip^ten; ® 



Title: Gives directions Evaluator (Coop.T,). 
Connaunication~Blue Date(s) Passed ^ 



09808 In b th verbal and written communications the intern will give clear and 
concise directions to students so that they can take appropriate action. 
Clear and concise directions involve: 

a. Speaking clearly enough for all students to hear when 
giving verbal directions. 

b. Making written directions as brief as possible. 

c. Planning and sequencing directions before giving them. 

d. Evaluating the resultant smoothness with which students 
act after receiving directions. 

To the Intern : Evaluation of this objective will be based on oral and 
written instructions you give to students. 

I. Oral Directions . 

No advance preparation is required for this portion of the objective. The 
cooperating teacher will determine by going over your lesson plans which 
session he would like to observe you in giving oral directions to your students. 

!!• Written Directions 

!• Invite your cooperating teacher 2-*3 day^: before you would like him to 
evaluate you on giving written directioas. 

2« Present a lesson of no more than one class period In length In which 
you give written directions to students. The directions should require 
students to act; e.g.» how to change a typewriter ribbon^ how to replace 
a blade in a saw» or how to thread a sewing machine. 

3« Give students a chance to ask questions after they receive the 
written directions and before they begin work. 

A* Collect (if a physical product resulted) or tabulate (if a visible 
action resulted) after the students have had time to follow your 
directions. Give these results to your cooperating teacher. 

To the Evaluator : Be in the classroom when the intern tells you he will 
be doing Part II. Written Directions. You may evaluate Part I. Oral 
Directions at anytime you wish noting from the lesson plans the Intern 
turns in to you exactly when he plans to give oral directions. Evaluate 
by criteria below: 

VeT^)al. Written 
Number of students in class: . 

Yes No Yes No 

A» Giving Di ^ actions 

!• The intern can be heard clearly by those 
foe whom the directions are intended? 
Criteria: Evaluator can hear and no 
students ask intern to speak louder. 

2« The directions are as brief as possibl^'i? 
Criteria: for a 30-mihute lesson» verbal 
directions should not exceed 5 minutes. 
Written directions should not be more than 1 1 

2 pages in length. Also, intern shouldn^t 



3. The directions are sequenced? Criterion: The 
directions specify the sequential steps to be 
taken In completing the task. 

4. t^ioTc the intern began giving directions 
verbally, the intern had all the students^ 

«r ten don? Criterion: None of the students for 
vhoa the directions are intended are talking to 
««ch other or working on other tasks. 

5. The students respond by doing what the directions 
caII for? Criterion: At least all but one 
be^inn to do what the directions ask. 



Verbal 
Yes No 



Written 
Yes No 



All yet'a arc required for a pass on Part A. 
B» Scufients' Questions After Directions 



.PASS 



REDO 



Tji^ly the number of questions according to the following categories which. 
t^e students ask after the directions are given. 



Verbal 



Written 



1» h\;:;:ber of questions raised by students which 
call for a verbatim repeat; e.g., •'What did 
you say?" 

2. Kurber of questions ' raised by students which 
»iv;nal a lack of understanding; e.g.» "I 
dcn'c know what to do," 

3. Questions that call for further information 
which according to the evaluator^s judgment 
•hould have been included in the directions; 
• .g., "How much time do we have?" 

4. All other questions. Questions which according 
to Che evaluator*s judgment did not need to be 
Included in the directions that signal special 
Interest on the part of students; e.g., "Can 

vc tvpe more than one letter if we want?" 
^^--rstloas which are unrelated to the directions 
jtvcn; e.g., "May I have a pass to see my 
counselor?" 



(Tally of questions) 
(on item does \) 
(not count towards ) 
(objective pass. } 



Uftft th^n a total of 3 questions from Part B 1-3 and the rest of the , 

•tuslcnttt on task while intern answers those questions are required 

for m pass on Part B PASS REDO 
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These are complex test items, necessary to measure a 
complex performance. This format has evolved out of the 
examination try-out. The greatest challenge was to include 
the necessary detail and yet xeep each item to one sheet 
(front and back). Earlier versions were longer, and the 
sheer size of the examination created many anxieties in both 
students and faculty. 

Two other points concerning format should be noted. 
First, every test item in the entire VAE Pre-certification 
Program is a public document.' (This is a radical departure 
from traditional programs and even from some competency-based 
programs.) The test item format reflects this principle. 
The student is aware of the exact directions and scoring 
procedures being used by the evaluator. And, likewise, the 
evaluator knows the student has this information. These 
details of the assessment process are viewed as further 
amplification of the criteria in the performance objective. 

The public nature of the test items presents few problems. 
The goal, in most CBTE programs, is to write terminal per- • 
formance objectives which demand higher-level performances 
(e.g., actual classroom demonstration of a skill), rather 
than simple recall. It is primarily with recall items that 
one worries about students knowing the test. Still, one 
could argue that with some recall items (definitions, for 
example) the exact answer can be memorized, since that is 
the learning task desired. Other recall items require a 
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sample response which represents the larger domain of knowledge. 
In these cases the precise questions are not provided in advance 
but only the characteristics of the questions. This situation 
occurs only a few times in the VAE Pre-certification Program. 

The second comment relevant to format concerns the question 
of fairness to students. In a fairly large program, one is 
confronted with a variety of evaluators— more than one pro- 
fessor teaching a course, many cooperating teachers in the 
public schools. It is important that all students* performances 
be measured consistently. Writing the test items in a detailed, 
standard manner provide some assurance that most students are 
being assessed objectively. 

Estabi : ~>hing Reliability 

Proponents of CBTE are being asked to prove that their 
certification recommendations are based upon valid, reliable 
measures of competencies. This is, at times, frustrating since 
these same, legitimate demands are seldom, if ever, made upon 
the traditional teacher educators. Nevertheless, the request 
is appropriate. The processes will be described here which 
are being used to determine the reliability of the VAE Pre- 
certification test items. Again, the intern teaching examination 
will be used as representative of the other work which is still 
in progress. 
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The state of the art of determining the reliability and 
validity of objective-referencpd test items is in its infancy. 
Authorities are just now experimenting with various techniques. 

The general process by which the intern teaching test's 
reliability was determined as follows: 

1. Identify the ability of the measures to produce 
the same assessment among raters of either a 
teaching performance or a product which is used 
in teaching (inter-rater reliability). 

2. Identify the capability of specific criteria 
questions used to determine adequacy of per- 
formance to produce the same answers among 
raters (inter-rater reliability). 

3. Identify the ability of measures to produce 
the same results on several performances by 
a given student when growth is held constant 
(trial-by-trial agreement). » 

This approach to reliability is particularly relevant to the 

tests written in the specific format chosen for the VAE 

examinations. 

The test was administered to all 43 of the intern teachers 
during the spring of 1973, and a total of 1688 observations 
were made to test the reliability of the 24 intern teaching 
test items. Each item for each student was rated by at least 
two persons. Raters were classified as either "prime" or 
"informed." Prime raters were either college supervisors or 
cooperating classroom teachers. Informed raters were all VAE 
faculty or staff. If faculty, they were not in the specific 



**Herschelmann, Kathleen M,, "Determininc the Reliability 
of a Criterion Referenced Intern Teaching Examination," 
unpublished doctoral dissertation, Wayne State University, 1973, 
p. 4. 
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curriculum area of the student; if staff, they were project 

research assistants. All raters were trained in the exact 

process to be followed. All data collection took place within 
g 

one month. 

Reliability was set at 84% for inter-rater agreement on 
tests and one disagreement or less for inter-rater agreement 
on items within tests. Reliability was set at 90% for trial -by- 
trial agreements. 

A process such as this can answer several questions for 
a faculty: 

1. Which tests are, in fact, reliable? 

2. Which tests need to be changed? 

The general try-out to get the reliability data leads to a 
wealth of additional data from the constituency which leads to 
recommendations for change. These recommendations may well 
relate to topics such as: 

1 . Format 

2. Understanding of criteria 

3. Common error points 

4. Data collection procedures 

5. Recycling situations 



The exact design of this study was developed by Jason 
Millman of Cornell University. The details of this procedure 
are found in the Herschelmann dissertation. However, in Appendix A 
of this paper, sections of the dissertation relevant to the design 
are reproduced. 

^^Herschelmann, p. 46. 

IG 
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6. Techniques to insure consistent evaluation 

7. Changes in performance objectives 

8. Ha.iJling irrelevant situations which occur during 
the student's performance 

9. Identifying relevant performance conditions which 
are not reflected on the test 

10. Scheduling of the test^^ 

Data on topics such as the above can well prove to be 
just as important as the reliability data itself. Any subse- 
quent study of this type should provide for the collection of 
such feedback for this gives the staff the direction for change. 
The reliability results simply tell you what must be changed. 

Establishing Validity 

The question of validity again creates a methodological 
dilemma. Ultimately, the validity of the test items stems 
from the validity of the required performance within the 
teacher training program. This validity should relate to the 
performance's effect upon pupils. Research must supply this 
answer. In a more limited context, the validity relates to^ 
the match of the item with the performance objective and the 
match of the item with previous instruction. This was the 
basic process used in the VAE program. 



Herschelmann, pp. 77-79. 
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The VAE validity study involved gathering judgmental 
data from three types of persons— the intern teachers, the 
cooperating classroom teachers and the college supervisors. 
The cooperating classroom teachers and college supervisors 
were asked three questions in relation to the test for each 
performance objective: 

1. Do teachers need the skill (s) demanded by this 
objective? 

2. Can your Intern perform the skill (s) needed for 
mastery of this objective outside of the test 
situation? 

3. Is the level of difficulty required by this objective 
appropriate for what is really needed in teaching? 

The intern teachers were asked: 

1. Were you taught, in your education courses, the 
skills needed for mastery of this objective? 

An Item was determined valid if 75% of the respondents 

12 

answered "yes" to each question. 

More detailed feedback was gathered by asking intern 

teachers to fill out an instrument in the Intern Teaching 

Seminar. The cooperating teachers and college supervisors 

13 

were interviewed using a structured interview form. 



"^Effective Feedback Associates, "Validity Analysis and 
RevisionsSuggestions for Intern Teaching Exam," Unpublished 
Report, June 30, 1973. Effective Feedback designed and 
implemented the entire validity study. 

^^See Appendix B for copies of these two instruments. 
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Pre-Intern Field Assessment 

Student assessment in a CBTE program is often more complex 
than in a traditional program; there are more evaluators, more 
settings and often more complex performances. Turner has cited 
three types of contexts for training and assessing student 
performances— the symbolic context (i.e., the college classroom), 
the simulated context (i.e., laboratory settings using audio- 
visual aids) and the "work" context (the public school class- 
room).^^ This section deals with assessment in the work context. 

The previous examples have dealt with the VAE Intern 
Teaching Examination. This takes place primarily (but not 
totally) in the public school setting. In this situation the 
cooperating classroom teacher is viewed as a part of the 
university staff, has regular contact with the university 
supervisor and works with the student on a daily basis. Hope- 
fully, this teacher also has had a long-term contact with the 
department. Because of this close relationship, the assessment 
procedures are identical to those followed on campus. 

The real issues regarding assessment in the "work" context 
seem to relate to the measurement of pre-intern teaching field 
experiences. The design of this part of CBTE programs vary 
widely. The VAE design is that of a field-oriented program 



'^Turner, Richard L., "Evaluating the Validity of Assessed 
Performances: Methodological Problems," a paper presented at 
the annual meeting of the American Educational Research Associati 
New Orleans, 1973, p. 2. 
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(as opposed to a field-based program). This means students 
have early field experiences which are carefully integrated 
into the campus instruction, but they are on a limited basis- 
three to four full days or six to seven half days per quarter. 
The cooperating classroom teachers may or may not be involved 
in the intern teaching program at that time. Nonetheless, 
their involvement in these early phases is more limited than 
that of the intern teaching supervisor. 

The classroom teacher still has the sign-off on the 
acceptability of student activity in the field in spite of 
this limited time. The products of these experiences are 
typically incorporated into a total campus test iteiP. 

The procedures are less rigorous than those of a full- 
scale test item. Appendix C is an example of a Pre-intern 
Field Experience Checklist.^^ This document includes: 

1. A statement of each field activity 

2. The required evidence of completion 

3. Supplementary lists providing additional 
criteria detail 

«< 

4. Space for teacher sign-off 

These documents have been developed jointly with representative 
classroom teachers and are supported as evidence of a student's 
field work. They should be viewed as a part of the overall 
VAE assessment and evaluation system. The field activities 
are key learning experiences which demand a type of quality 
control . 

^ ^McMillan, Marian, "The VAE Pre-Intern Field Experiences: 
A Model for Competency-Based Teacher Education Field-Oriented 
Programs," (Detroit, Michigan: Vocational and Applied Arts 
Education, 1974), pp. 14-18. 
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ASSESSMENT DATA MANAGEMENT 

Data Collection 

The assessment procedures described generate a great deal 
of data. In the VAE program the instructional system is supported 
by a computerized management information system. The primary 
goal of the instructional management subsystem "is to eliminate 
as much manual record keeping as possible on the part of the 
faculty but at the same time provide information on the status 
of each student as he progresses through the program. "^^ 

Faculty submit student test data in one of two ways— the 
Objective Record Form (a computerized grade book) or on an 
Optical Mark Recognition Card (OMR).^^ All intern teaching 
data is turned in on the OMR Card. 

Reports Available 

There are a variety of reports which are available to VAE 
faculty as a result of the test data submitted. These include: 

1. Status Reports with; 

a. a list of objectives for a class 

b. students passing objectives 

c. date passed 

d. number of recycles 

2. Histograms including: 

a. number of people passing the exemption and 
exit tests 

b. minutes to complete an objective 

c. recycles per objective 



Neuhauser, Charlotte L., "The Design and Implementation of a 
Management Information System to Facilitate the Functioning of a 
CBTE Program," Paper presented at the annual meeting of the 
American Educational Research Association, Chicago, April, 1974, p. 13. 

^'''see Appendix B for a sample of each of these two data-collecting 
devices. 
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3. Curriculum Status Reports with; 

a. a list of students in curriculum area 

b. all objectives completed 

c. date of completion 

d. objectives not completed 

e. recycles 

f. flagged student problems 

4. Student Status Letter includes: 

a. all objectives to date 

b. objectives completed 

c. date of completion ig 

d. special message indicating next steps 

PROGRAM EVALUATION 
Formative Program Evaluation 

To this point, the model described has dealt with 
student assessment, but there is another major part— progrc'n 
evaluation. This is the attempt to collect and use data for 
the revision of instructional materials, processes and content. 
Within this framework, there is one major feature of the formative 
evaluation phase which has been systematically used— the Student 
Content Evaluation. 

Throughout each quarter, students answer the questions 

listed in FIGURE 2 for each and every performance objective 

they complete. This data is then keypunched from the form and 

entered into the VAE computerized data bank. The resulting 

Content Evaluation Report provides information for revision of 

19 

performance objectives and teaching methods. 



^^Neuhauser, pp. 16-17. Samples of each of these reports are 
taken from this paper and found here in Appendix C. 

^^Neuhauser, p. 20. 
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FIGURE 2 
CONTENT EVALUATION QUESTIONNAIRE 

1. Did prerequisite courses, if any, give you an adequate foundation in 
meeting the objectives of this course? 

Yes No No Prerequisite Requirements 

2. Was adequate time, materials and facilities provided for you to 
achieve this objective? 

Yes No 

3. How meaningful were the assignments in relation to the mastery of this 
objecti ve? 

Assignments were Assignments Could have Uncertain 

helpful in attaining helped some mastered objec- 
mastery but could be tive without 

improved completing 
assignments 

4. How appropriate were the methods (kits, films, etc.) used in conveying 
the instruction to master this objective? 

Appropriate— Were OK but Were Uncertain 

I liked them I prefer other Inappropriate 

methods 

5. Did the exit test accurately measure the behavior scjght in ihis 
objective? 

Yes No 

6. Do you feel you have really mastered this objective? 

Yes, feel very Yes, but need Yes, but need Did not pass 

competent some reinforcement much more exit test 

instruction 

7. In hindsight do you think you could have passed the exemption test 
on this objective at the beginning of the quarter without receiving 
instruction? 

Yes No 

8. Do you feel this objective is essential to your teaching preparation? 

Yes No 
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A growing. bank of data of this type creates an invaluable 
base for decision making. The decisions related to delivery 
systems are made by individual professors in the VAE program 
design; however, if the professor chooses, questions such as 
the following could be answered using this data: 

1. Should the pacing of instruction be altered? 

2. Should new instructional materials be identified? 

3. Did students feel prepared for the instruction 
fn this class? 

The faculty can use the data to make performance objective 
revisions. Revisions can become apparent from this data result- 
ing from topics such as: 

1. Student feeling of irrelevance 

2. General student knowledge before instruction 

3. Need to provide a firmer base of prerequisite skills 

Summative Program Evaluation 

The design of the follow-up procedures for the VAE Pre- 
certification Program is still fn progress. This is a longi- 
tudinal development process that has three key phases: 

1. Follow-up instrument design and use 

2. Observation of graduates 

3. Testing pupils of graduates 

First, is the design of a follow-up instrument. This 
phase in itself has a longitudinal development plan. The first 
step is to interview graduates (using trained interviewers) 
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using an open-ended instrument. The major topics covered in 
this interview are: 

1. Student perceptions of the program (including 
reactions to the design, goals and activities) 

2. Impact of the program on student attitudes 
toward teaching as a career 

3. Impact of the program on their own teaching 

The initial data drawn from these interviews will provide 
the basis for the gradual revision of the instrument. After 
use with three to four graduating classes, it is planned to 
convert the instrument Into one suitable for use as a mailed 
instrument. At this point, the follow-up could become a 
regular part of the program, existing with little special 
support work. 

However, this type of follow-up gathers only secondary 
evidence. In order to actually measure the continuing effect 
of a program, one should do two things: 

1. 0'i;erve the graduate in a teaching situation. 

2. Measure the effects of the graduate's teaching 
on pupil behaviors. 

While this has not been attempted to date, instruments and 

procedures are being devised which will facilitate this phase 

of the follow-up. 

Bastcally the plan involves testing groups of students 

before and after a semester or a year's work. Between testing 

periods the teachers are systernatically observed. The 
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observation system will be a modification of the APPLE System 

(Anecdotal Processing to Promote the Learning Experience). 

This is a system developed over the past five years by Nadine 

Lambert and her associates at the University of California 

at Berkeley. It is not a category system, and it approaches 

the task by simply recording the observed behavior of the 

teacher and selected target students representative of the 

class. Subsequent encoding of the data categorizes the 

observations using a lexicon developed out of observations 

in that particular subject matter area. 

These observations can achieve two goals: First, they 

can be used to determine if the teacher is, in fact, using 

those competencies which were a part of the VAE Pre-certification 

Program. Second, they become a part of a detailed research 

design, which when connected with the student pre- and post- 

te t scores, can identify those teacher behaviors which do 

21 

affect pupil performance. 

Not until this process is completed will one really have 
a complete assessment and evaluation model. This summative* 
evaluation facet should take approximately three years to 
complete. 

^^Lambert, Nadine and Hartsough, Carolyn S., "Instructions 
for APPLE Observers in the Beginning Teacher Evaluation Study," 
(Berkeley: University of California, 1974). 

^Vor a complete description of this research design see "A 
Proposal to Determine the Effects of Teacher Performance on Pupil 
Growth" by Fred S. Cook, Rita C. Richey, Vivian Howell, Patricia 
Elias and Frederick J. McDonald, (Detroit, Michigan: Wayne State 
University, 1974). 
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One of the great advantages of this latter research 
approach to summative evaluation is that it will also produce 
data which can allow one to legitimately make hypotheses 
regarding which competencies are valid (validity being affecting 
pupil growth). In this way, the data base out of an assessment 
and evaluation model will provide input for the competency 
identification phase of designing CBTE programs. 

SUMMARY 

This is a comprehensive plan, the majority of which is 
operational. It is, however, still operational in a context 
where, if CBTE were to fail, it would be because of attacks on 
the assessment system. The assessment and evaluation system 
(along with the initial competency identification) is the most 
crucial aspect of CBTE; it is the most vulnerable; it needs 
the most supporters of CBTE. 

The following flowchart is a graphic summary of the system 
as a whole. See FIGURE 3 on the next page. 
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FIGURE 3 
A FLOWCHART OF 
THE VAE ASSESSMENT AND EVALUATION SYSTEJl 
OPERATING PROCEDURES 
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Collect Collect 13 

Observation Follow-up 

Data From Data From 

Grads & Their Graduates 

Pupils ' 1 ' 




YES 



The VAE Assessment and Evaluation System is a comprehensive 
plan which is designed to meet goals relating to campus and field 
assessment of students prior to, and after, instruction, as well as 
systematically collecting and using data to make short-term and 
long-range revision in the program based upon the performance of 
graduates and their pupils. 
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APPENDIX A 

PROCEDURES FOR DETERMINING TEST RELIABILITY 

Determining the Research Design 

In determining the research design to be used with the 
first administration and evaluation of the VAE CRITE, the 
major consideration was to develop a design which would lend 
itself to a criterion referenced examination* As Jason Millman 
of Cornell University had worked v/ith James Popham of UCLA in 
developing strategies by which criterion referenced examina- 
tions could be produced^ administered* and evaluated* the 
investigator asked Millman's assistance in determining the 
research design for this study* 

Millman suggested that a inay of determining the relia^ 

bility Cgeneralizability)^ of the 24 tests in the VAE CRITE 

would be to use both— • 

K rater agreement (two or more raters evaluating 
a live performance simultaneously or a written 
performance without inter-rater consultation) 

2. trial -by-trial agreement (one rater evaluating 
a test on two occasions in a short time period ^ 
holding "growth" constant by not giving the 
intern feedback until completion of the second 
trial 

The 24 tests were split into 3 groups of 8 tests each. 
Test group selection was based on time involved in taking 
the tests and whether the tests would be evaluated by two 
raters at one session or by one rater in a trial -by-trial 
situation* This was in accordance with Millman*s suggestion: 

Hext prepared by Jason Millman in consulting session 
with investigator* Cornell University* Ithica, New York* 
April 23* 1973* p, Z. 
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...divide the 24 tasks into three groups of 8 each- 
calling them Group A, Group B, Group C. Remember, 

B> C refer to collection of tests and not to 
groups of interns. I would suggest that the grouping 
of the 24 tasks NOT be random, but instead be as 
follows: 

Find the time-consuming ones and don't overload one 
of the groups—split up. 

Find the tasks you want to do rater ^agreement on and 
split up. Same for trial-by-trial 

Rater Agreement 

Twenty-one of the 24 tests were submitted to rater 
agreement design^ A college supervisor and a cooperating 
teacher could jointly evaluate an intern on a test or a set 
of tests. A second alternative prov1l-»i that one of these 
prime raters could join with an inf craned rater to do an 
evaluation. The research design assigned specific prime 
raters and informed raters to each Intern for each objective 
or each set of objectives. In the event a prime rater needed 
a substitute for his rating partner, an Informed rater was 
sent in as the substitute so the evaluation could be performed. 

■' ♦ 

Trial -by-Trial Agreement 

Three of the 24 tests were subjected to trial -by-trial 
evaluation. The three tests were chosen because the cooperating 
teacher could evaluate the tests in two of the Intern's classes 

^Ibid. > p. 2. 

*two or more raters evaluating a live performance simultaneously 
or a written performance without inter-rater consultation 
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holding "growth" constant as the intern would not receive 
feedback until both situations had been evaluated. These 
tests were: 

09210 Manages supplies and equipment 

09211 Organizes routine procedures 
09219 Responds to evaluation 

These three objectives were among the tests in Group C. 

Raters Training Session 

The cooperating teachers, college supervisors, informed 
raters, and the rest of the VAE faculty were Invited to par- 
take in the raters training session held on May 5, 1973. 

The full -day session had as its major objectives: 

1. train each rater to administer and evaluate 
the tests for which he was accountable, 

2. provide all VAE faculty attending the experience 
in organization, administration, and evaluation 
of the VAE Criterion Referenced Intern Teaching 
Examination, 

3. provide time for all raters to set up their 
schedules with their rating partners so all 
rating dates would be definite by the end of 
the day. 

. . •# 

Rater Training 

Each rater was given a binder which held: 
. 1. a copy of the day's agenda, 

2. a complete set of the tests he was to administer 
and evaluate, 

3. directions for scoring the examination. 
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4. a calendar of suggested dates when the intern 
might hand In his tests, 

5. a suggested combination of objectives to enable 
the rater to evaluate the objectives in fewer 
classroom visits, 

6. an evaluation time sheet giving approximate 
time for administering and evaluating each 
objective, (Each rater was also asked to 
add his actual rating time.) 

7. an evaluation schedule listing all raters, 
interns, and the objectives assigned to each, 

8. a supervisor's time sheet listing approximate 
number of visits and total time to complete 
all evaluation visits, 

9. an informed raters' assignment sheet listing 
all interns and objectives for which they 
were accountable, 

10. a tax form to be filled out and submitted at 
the end of the day to receive the training 
stipend. 
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Objectives-Based Measures 3] 



I^trectlons and Purpose ; 

VAE is asking you to evaluate objectives-based measures administered during your intern teaching 
assignment. Your comments will influence any revisions which must be made before Fall, 1973. 
Answer every question. Thank you for your assistance. 



1. ObjectlveOCnCIIZIID Objective Title . 

» ft -a ^ ff t 

2. Intern Assignment (check one) : High SchoolO Junior High CD Other (spec if y)_ CD 

.J" o •* 

3. Name the courses you are teaching :^ ^ . 

I O CZI. 

X 



I _ i^, 



4. Were you taught, in your education courses, the skills needed for mastery of this objective? 

a. YesC^ SoraewhatC^ 5? . ' 

b. WHERE? 519iCr3 5925 (or 3731 & 3015) d Methods CoursesQ Other CD 

jw 4.i» ** , "lo 

5. How often have you used the skill needed for Kc.stery of this objective during Intern 
Teaching? (aside from this situation) 

Of ten cm Sometimes Hp Never CZJ 
Comment: 



6. Are there other facets of this skill which should have been meas; ; .ed? Yes CI No O 
Explain: . . ^ ! 



7. Are there any undesirable features of this measure? Yes CD No CD 
Explain: 



8. What are the best features of this measure? 



r L_iL_j„j f" I I ■ i I — r~i — t I I — f — I I I I I I i i I — I — I I 1 1 



Date 

Interviewer 



Interview Questions Por Cooperating Teachers/ 
Seminar Leaders/College Supervisors 



1. Name ' V_ School 



2. Subjects to v/hich the Cooperating Teacher is assigned 
Business Education Industrial Art 



Family Life Other (please specify) 

3# Number and name of objective being discussed 
# ' Name 



4. Did the Intern master this objective? Yes .^.^..^ No 



.5^ Can your Intern perform the skill(s) needed for laastery of 
this objective outside of the test situation? 

Definitely ' Unsure Not at all 

Comment; 



6^ Do teachers need the skill(s) demanded by this. objective? 
Yes . Somewhat ^..^^^ No . 

Comment: 



Rir 
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Is the level of difficulty required by this objective 
appropriate for v/hat is really needed in teaching? 

Yes ______ Somev/hat ' No 

Comment: 



V/hat should be added to the requirements? That is, are 
there any skills like this one that teachers should display 
in-order to teach? 



Are there any xmdesirable features of this measurement 
item? 



What is the best feature of the measurement item? 



TO BE ASKED AT END OP INTERVIE'// ONLY: 

Do the total measures (all that you evaluated) cover enough 
to convince you whether or not the Intern hcL£ the desired 
skill to be a teacher? 

Yes Somewhat No • 

Comment: 
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APPENDIX C 



Vocational and Applied Arts Education 
Pre-Intem Field Experiences 
Phase I - Check List 



Directions: When designated experiences have been completed in the 



school, the pre-intern will record the date in the 
appropriate space, and the classroom teacher will indi- 
cate approval by placing his or her initials in the 
appropriate space. 



Field Experience 



1. Interview a cooperative education coordinator, 
placement counselor, teacher in job preparation 
program, or a person giving leadership to career 
education in the school. 

a. What is the overall plan for career 
preparation and placement of students? 

b. How does the career education program 
operate in the school? 

c. What ts the "cha r. of conmand" of people 
responsible for career education in the 
school district? 

d. What plans are in the offing for changes 
in occupational preparation? 

Evidence of completion: 

a. A list of the system-wide "chain of 
command" for career education 

b. A written summary of the career 
education activities being Implemented 
or planned by each career education 

person In the school 



2. Interview a classroom teacher to obtain an 
overview of the tasks and responsibilities 
required of a teacher. (See#3.) 

Evidence of Completion: 

A brief description of specific 
responsibilities of a classroom teacher 
(classroom responsibilities, contractual 
responsibilities, personal commitment). 
(See supplement for guidelines.). . . . 



Date Classroom 
Completed Teacher Initials 



Date 
Completed 



Classroom 
Teacher Initials 



37 



35 

Phase I, Page 2. 



Field Experience 



3. Collect and surrjnarize information relating to 
an in-school program or course using four of 
the following sources: 

a. Review a curriculum guide for a specific 
program (i.e.. Business and Distributive 
Education, Child Growth and Development, 
Drafting, Electronics) 

b. Interview VAE curriculum coordinator or 
department head 

c. Interview teacher (See #2.) 

d. Interview students for reactions to the 
program 

e. Attendance at a curriculum planning 
committee meeting 

f. Observe at least one class session 
of a course being reviewed 

Evidence of Completion: 

Appendix material to be included in report 
in VAE 5191 in connection with the evaluation. 

(See supplement for guidelines.) . . 

uate Classroom 
Completed Teacher Initials 

4. Participate in a minimum of five of the 
following classroom activities: 

Check List 

a. Assisting on projects 

b. Escorting students to lunchroom 

for a given class * 

c. Answering questions 

d. Correcting papers 

e. Checking supplies in and out 

f. Acting as a small group leader 

g. Assisting in a demonstration 

h. Introducing audio-visual materials 

i. Taking roll 

j. Others 

Evidence of Completion: 

A check list identifying the experiences 

in which the pre-intern participated _^ 

uate Classroom 

Completed Teacher Initials 
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Phase I, Page 3. 



Field Experience 



5. (Optional) Arrange for a field trip with the 
cooperative education coordinator to visit a 
work station. 

Evidence of Completion: 

A brief written overview of the field 

experience ^ 

Date Classroom 

Completed Teacher Initials 



Has the pre-*intern made himself convenient to your 

schedule? Yes No 



General Comments: 



Approximate time for Phase I: 6 half days or 3 full days 
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Supplement to 
Field Experience Requirements 
Phase I 



Guidelines for Item #2 , 

1. Classroom responsibilities 

a. records keeping 

b. management of supplies 

c. order 

d. clean up 

e. safety precautions 

f. others 

2. Contractual responsibilities 

a. time schedules 

b. attendance at meetings (type and frequence of meetings) 

c. curriculum development 

d. others 

3. Personal commitment 

a. individual attention to students 

b. adequate arrangements for make-up work 

c. attendance at school functions 
T rommuntty activity 

e parent contact 

f. positive comments to individuals 

g. others 

Guidelines for Item #3 > 

K Review of curriculum guides 

a. Does it give an overview of the purpose of the guide? 

b. Does it explain how the guide is to be used? 

c. Does it present an overview of subject content? ^ 

d. Does it include the component parts for writing 
performance objectives? 

(1) Do objectives make provisions for occupational 
preparation? 

(2) Does it reflect provision for progression from unit to 
unit? 

(3) Are objectives stated so students and parents can 
understand them? 

(4) Are objectives realistic in terms of achievement? 

e. Learning experiences 

(1) Does it reflect a variety of experiences? 

(2) Does it present experiences appropriate for age level? 

(3) Does it reflect current societal needs? 

(4) Does it include experiences for evaluation? 

(5) Do?s it allow for s^lf-evaluation by students? 

(6) Does it include current emphasis on career education? 
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Supplement 9 page Z. 

f. Summary: Personal reactions 

(1) Did this help you with an overview of the course, 
units, or curriculum area? 

2. Interview with curriculum coordinator or department head: 

a. Does your school have an advisory committee to make 
recommendations to the Vocational Department? If "yes," 
what areas are represented from business school, admini- 
stration, etc.? 

b. How often does the department make a "self-evaluation" of 
the program, the instruction, the facilities? 

c. How are the present curriculum programs up-dated and changed 
to include current economic, business and technical practices? 

d. Does the entire department work together on program 
development and improvement? How? 

e. How do you evaluate individual teachers? 

3. Interview with students: 

What baste skills do you feel you have acquired from this 

course in ? 

What were some of the resources you used in the course 
(films trips, films, newspapers, periodicals, reference 
books, etc.)? 

How were the current social, economic, or technical changes 
in society interwoven in the course? 

In what way were students involved in the planning of units? 
In evaluating units or the total course? 
What were some of the different learning experiences you 
had in the course? Which did you feel were the most 
stimulating? Worthwhile? 



a. 
b. 



c. 
d. 
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